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Brief Three 

Robust Implementation of MDC: Teacher Perceptions of Tool Use and Outcomes 

September 2012 


Introduction 


The Bill and Melinda Gates Foundation has invested in the development and dissemination of high- 
quality instructional and formative assessment tools to support teachers’ incorporation of the Common 
Core State Standards (CCSS) into their classroom instruction. Lessons from the first generation of 
standards- based reforms suggest that intense attention to high quality instructional tasks (City, Elmore, 
Fairman, &Teitel, 2010; Hiebert and Carpenter, 1992; Hiebert and Weame, 1993; J ones et al, 1994) 
and use of formative assessments embedded in those tasks (Black et al, 2004; Clarke and Shinn, 2004; 
Fuchs, 2004; Tunstall, 1996) are essential if teachers are to meet the demands of the CCSS. 


Experts from the Shell Centre have developed a set of Formative Assessment Lessons (Lessons) for 
secondary mathematics teachers to facilitate CCSS- based student mathematics learning and provide 
teachers with feedback about student understanding and mastery. The tools are designed to target the 
“instructional core” by: 

• Raising the level of content 

• Enhancing teachers’ skill and knowledge about instruction, content and formative 
assessment 

• Catalyzing student engagement in their learning so that they will achieve at high levels 
(Elmore, 2010) 

These tools have been in use for two years (2010-2011 and 2011-2012 schools years). 

In This Brief 

This brief highlights and assesses the status of elements of robust implementation of the MDC tools, 
which are represented by the small blue circles in the Theory of Action ( see Figure 1) . These six 
indicators, which fall into two main categories - Teacher Beliefs and Knowledge and Classroom 
Changes- are instrumental in understanding teachers’ disposition towards the tools and their 
perceptions of how their instruction and student learning have changed as a result of their participation 
in the MDC initiative. Robust implementation should lead to several intermediate and long-term 
outcomes, among them Broad and Deep Instructional Change. We present our findings for this 
outcome as well. Figure 2 provides definitions for each Robust Implementation Indicator, as well as for 
the Intermediate Outcome of Broad and Deep Instructional Change discussed in this Brief. 
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Figure 1. Theory of Action 
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Figure 2. Definitions of Robust Implementation Indicators 
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Definition 


Teacher buy-in to instructional and curricular initiatives is central to the success of any 
new reform. Teachers need to believe that the initiative itself and its supporting 
structures will provide them with the tools to help their students achieve at higher levels. 


The use of the Formative Assessment Lessons requires teachers to adjust their math 
instruction to teach in fundamentally different ways than most teachers traditionally teach 
math, including the constant facilitation and assessment of student learning as opposed to 
providing direct instruction, allowing students to struggle to develop their own 
conceptual understandings of mathematics. 


In order for teachers to successfully use the Formative Assessment Lessons and increase 
student learning, they need to have a strong understanding of how to use them. Teachers 
need to know how to place the Lessons in the larger math units and discern and respond 
to students’ mathematical misconceptions. 


Once teachers know how to use Formative Assessment Lessons, they need to execute 
new pedagogical methods in ways that change instructional practice. 


Students must be responsive to, and engaged by, the new instructional practices in order 
for student learning to improve and for students to graduate from high school college 
and career ready. 


Teachers need to recognize the utility of the tools and personally perceive improvement 
in student learning prior to making a greater investment in the initiative. 


As Robust Implementation takes hold and deepens, teachers will exhibit significant 
changes in their pedagogy that will extend beyond the confines of the initiative and into 
general classroom practice. 


We provide a brief overview of data sources and then examine: 1) the three indicators of robust 
implementation related to teacher beliefs and knowledge; 2) the three indicators of robust 
implementation related to classroom changes; and 3) signs of broad and deep instructional change. 
Brief 3 doses with recommendations and questions. Throughout this brief, we use the following 
symbols to indicate: 


A status update of how teachers are using the tools - Descriptive only; no 
indication of whether the activity is positive or negative. 

^ Successful aspects of tool implementation. 



Issues LDC and participating sites need to watch for, such as challenges 
teachers are confronting or questions raised by the data. 


Data Sources 

Research for Action began conducting research on this initiative in its co- development year of 20 10- 
20 11 (Year One), and has continued this research into the pilot year of 20 11- 20 12 (Year Two) . This brief 
draws on data relevant to understanding teacher knowledge about and use of the MDC tools in their 
classrooms in Year Two (see Figure 3) . These data are a subset of Year Two data collection; the other 
briefs in this series examine other aspects of MDC and Literacy Design Collaborative (LDC) using 
additional data. 


Figure 3. Classroom-Level Data Sources 
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The number of teacher survey respondents remained almost the same in Year One and Year Two, only 
increasing by 13 respondents from Year One (83 respondents) to Year Two ( 96 respondents) for the 
following reasons: 

• Survey sample changed but did not significantly expand from Year One to Year 
Two. A number of factors constrained the growth of our survey and research sample. 

o In Year One, teachers in RFA’s four study sites and all other sites responded to the MDC 
teacher survey, for a total of six sites. In Year Two, the survey focused on teachers in 
RFA’s four study sites so that complimentary qualitative data could be used to accurately 
explain survey data. 

o With the attrition of a fairly large Year One RFA study site and after consulting with 
Gates Program officers and PD consultants, RFA added a new site that was in the early 
stages of MDC implementation, which contributed to a smaller survey sample. 


1 Teacher survey sites include returning Year One sites and one new site. One other Year One site did not return for Year Two, so were not 
included in the survey. 
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o Additionally, there were few new MDC sites to choose from since several new large sites 
j oining the initiative would not be up and running until mid-year. Some Year One MDC 
sites did not continue to implement the initiative in Year Two, while other sites had no 
room to expand their implementation of MDC because all of their district teachers were 
already involved. 

• Site MDC Implementation plans and contextual issues affected the number of MDC 
teachers. 

o One site included all of their math teachers in the MDC initiative in Year One; therefore 
there was no room to expand the initiative to new teachers, 
o Two other sites made participation in the MDC initiative optional, which meant that the 
initiative was not systematically implemented district- wide. In one site, there was a 
reduction in MDC teachers from Year One to Year Two. The other site was in the early 
stages of MDC implementation and involved small numbers of teachers, 
o Lastly, the fourth site decided to expand their CCSS tool implementation by adding the 
LDC initiative instead of expanding MDC to additional teachers. High teacher turnover 
in this site also proved challenging in retaining teachers in the MDC initiative. 

Of the 96 respondents, 50 (54%) were Experienced MDC teachers and 43 (46%) were New MDC 
teachers. (Three teacher sis not respond to the question) . 

RFA triangulated all of the above data (interviews, surveys and observations) to understand and 
measure the enactment of the six robust implementation indicators and the intermediate outcome 
indicator (Broad and Deep Instructional Change) in Year Two. 

Prior to Year One of the MDC initiative, the Gates Foundation provided “readying” professional 
development to teachers that focused on formative assessment strategies. In doing so, the Foundation 
prepared teachers to use Formative Assessment Lessons, which were still in development. The 
population of teachers who used the Lessons in Year One includes teachers who received the formative 
assessment “readying” professional development and those who did not. In this brief, teachers with at 
least two years of experience using the Lessons are referred to as “Experienced teachers.” “New 
teachers” are those who began participating in the MDC initiative in Year Two and only have one year of 
experience. Much of the qualitative and survey analysis in this brief focuses on both overall teacher 
perspectives in Year Two, as well as on differences between new- to- MDC and experienced teachers. 

Site- specific contextual issues can be an important factor in implementation of MDC, so when relevant 
we examine such issues as mediating factors for implementation. Also, when implementation 
challenges and successes have remained constant - or changed dramatically - from Year One to Year 
Two, we compare the results of these two evaluation periods as a lens for understanding the data. 


MDC Implementation Context 

District and network leaders created MDC implementation plans. They made decisions about whether 
to focus implementation in particular grades or content areas and about how many Formative 
Assessment Lessons teachers would implement. At the time of the teacher survey (March/ April 2012), 
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all teachers had taught an average of 4 Lessons. Experienced teachers had taught 4.5 Lessons on 
average, and New teachers had taught 3.2 Lessons. Site was also a factor affecting the number of 
Lessons taught. In one site, teachers had taught an average of 4.5 Lessons, while the number of Lessons 
teachers taught in the other three sites ranged from 3.5-3. 1 Lessons. 

When teachers were asked what the number of Lessons they would like to teach in the upcoming school 
year (2012-2013), the mean response was 6.3 Lessons (the median was 6 Lessons). Experienced 
teachers were more likely to report wanting to teach a higher number of Lessons - 7. 7 Lessons. On 
average, they reported wanting to teach 7.72 Lessons compared to 4.95 Lessons reported by New 
teachers. The largest percentage of both Experienced (28%) and New teachers (22%) reported wanting 
to teach 6 Lessons next year. There were no meaningful differences between the sites and the median 
number of ideal Lessons, which ranged from five to six Lessons. 


Elements of Robust Implementation 

As Theory of Action shows, when the conditions necessary to support tool use are in place, robust 
implementation emerges in two forms: Teacher Beliefs and Knowledge, and Classroom Changes. Our 
findings regarding each are summarized below. 


Teacher Beliefs and Knowledge 

This section examines the status of 
the three indicators of robust 
implementation related to teacher 
beliefs and knowledge depicted in the 
Theory of Action and highlighted 
here. 


Teacher Beliefs and 
Knowledge 


Classroom Changes 
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Robust Implementation Indicator: 

Teachers believe in the underlying principles of the MDC tools 


Teachers believe 
in the underlying 
principles of tools 


The use of the Formative Assessment Lessons requires teachers to adjust their math 
instruction to teach in fundamentally different ways than most teachers traditionally 
teach math. For example, when teachers use Formative Assessment Lessons, they 
facilitate their students’ learning of math as opposed to providing direct instruction. 
Teachers are also required to ask guiding questions and provide class time for students 
to persevere through the mathematical demands of the Formative Assessment 
Lessons, often by working collaboratively with classmates. 


© Teachers’ beliefs about math instruction aligned with the underlying goals of the 
MDC initiative. 

• All teachers surveyed reported that asking students guiding questions was an effective 
instructional strategy (see Figure 4). 

• A vast majority of teachers (99%) reported that peer-to-peer problem-solving is an 
effective way to strengthen students’ mathematical understanding. 

• The vast maj ority of experienced and new teachers reported that taking on the role of 
“facilitator” or “coach” was an effective instructional approach. 

• Experienced teachers ( 100%) were significantly more likely than new teachers (93%) to 
report that providing class time for students to persevere through difficult math 
problems is an effective method of strengthening students’ mathematical understanding; 
although, the results of this survey item are very positive overall. 


Figure 4. Teachers' agreement with the underlying principles of the Formative Assessment Lessons 
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**Differences between experienced and new teachers are significant at .1 level. 
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MEDIATING CONDITION 


Site-specific contextual factors affected teachers' beliefs in the effectiveness of some of the distinguishing features of the 
Lessons. For example, teachers in one site are in their first year of implementing a new curriculum. Although teachers 
reported that the new curriculum aligned to the Formative Assessment Lessons, they are experiencing some pacing 
challenges as they learn the curriculum. 


Robust Implementation Indicator: 
Teachers Exhibit High Levels of Buy-In 


Teacher buy-in to instructional and curricular initiatives is central to the success of any 
new educational policy or reform that plays out at the classroom level. Teachers need 
to believe that the initiative itself, and its supporting structures, will provide them with 
the tools to help their students achieve at higher levels and, in the case of the MDC 
instructional tools, that students will graduate from high school not only with strong 
math skills but also well- prepared for either college or a career. 

© Both experienced and new teachers are committed to making improvements in 
how they use the Formative Assessment Lessons. Teachers’ ability and willingness to 
gain facility in using the Lessons is very important to the sustainability of the MDC initiative. In 
addition to understanding the common structure of a Formative Assessment Lesson, teachers 
also need to gain facility in using individual Lessons, as each one requires teachers to 
understand different content and different student misconceptions. Promisingly, 98% of 
experienced teachers and all new teachers reported that they plan to make improvements to how 
they use the Formative Assessment Lessons next school year (see Figure 5) . 


Teachers exhibit 
high levels of buy- 
in to the initiatives 


Figure 5. Teacher Buy-in to the MDC Initiative 
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*** Differences between experienced and new teachers are significant at .05 level. 
****Differences between experienced and new teachers are significant at .01 level. 
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© Experienced teachers and new teachers are enthusiastic about Formative 

Assessment Lessons. Eighty- nine percent of teachers reported that they would like to receive 
more Formative Assessment Lessons next school year (see Figure 5). Ninety- six percent of 
Experienced teachers indicated that they look forward to teaching Lessons next year compared 
to 81% of new teachers. Though reports of teachers’ interest in both receiving and using more 
Lessons next year are positive, these findings suggest that as teachers gain more experience 
using the Lessons, they may become more interested in using them. 


MEDIATING CONDITION 


Once again, site-specific contextual factors affected teachers' interest in the receiving more and teaching more 
Formative Assessment Lessons next school year. All teachers in two of the four sites reported that they are looking 
forward to receiving more and teaching Formative Assessment Lessons next school. But in the other two sites, only 79% 
and 85% reported such enthusiasm, The significant minority of teachers who were not enthusiastic in these sites may be 
feeling overwhelmed by the demands of the new curriculum which was implemented recently, and challenges of 
including Formative Assessment Lessons in their pacing guides. 


Experienced MDC teachers were significantly more likely than new teachers to 
^ report that the Lessons had become an important part of their instructional 

practice. Eighty- four percent of experienced teachers reported that the Formative Assessment 
Lessons had become an important part of their instructional practice, while only 56% of new 
teachers reported that the Lessons were an important component of their instructional 
repertoire (see Figure 5) . 


Experienced teachers (94%) were significantly more likely to report that their 
participation in the MDC initiative was worth their time and effort it has involved 
than new teachers (79%) (see Figure 5). Participation in the MDC initiative required 
considerable time commitment from teachers. They had to participate in professional 
development that was often held after school (across two consecutive days at a time), prepare 
Formative Assessment Lesson materials to use in the class, and spend at least two to three days 
using the Lessons with their students. After participating in the initiative for at least two years, 
experienced teachers may perceive a larger pay-off to their participation than teachers who are 
new to the initiative. 



While most teachers reported high buy-in to the MDC initiative, some teachers 
found it challenging to integrate the Formative Assessment Lessons into other 
curricular initiatives. Teacher interview data indicated that most find value in using the 
Formative Assessment Lessons, while a significant minority reported being overwhelmed by the 
many curricular initiatives in their district or the Formative Assessment Lessons’ lack of 
alignment with the curricula and state assessments. 


9 | B r i e f Three: MDC 


High buy-in 


We [moth colleagues] oil felt that there was value in it. There was no argument or discussion; we all agreed that we 
needed a different approach because of the students we are getting now [with low math skill levels]. ..This [MDC] 
was very refreshing for us to find out what a Formative Assessment Lesson has to offer to us as teachers. It was 
very interesting to us , but at the same time , we have to have the open mind that we have to change as teachers if 
we want to really make this successful. The change makes sense to us and we were ready to change. We could not 
go on with the same model we were following. (Experienced high school math teacher) 

Concerns about competing initiatives 

It's hard to tell right now [if my participation in the MDC initiative is worth it]. We have just moved to a new 
curriculum and so , adjusting to that and this is not just like moving to a different book , it's a totally different way 
that things are laid out , how things are lined out and we are expected to follow it really closely. So it's really tough 
to see how things are planning out time wise and with the trimesters it's hard because you don't have as much 
time , so I am waiting to see ; the cost benefit. (Experienced high school math teacher) 

Concerns about lack of alignment 

It [MDC] differs [from my approach to teaching].../ just don't have the time to do all 4 components , and I'm 
restricted with the time. I have a course curriculum to fill, and I have to prepare my students for the integrated 
algebra regents , and I feel that it's still not aligned. (New high school teacher) 


Robust Implementation Indicator: 
Teachers know how to use the tools 




Teachers know how 
to use the tools 




In order for teachers to successfully use the Formative Assessment Lessons and 
increase student learning, they need to have a strong understanding of how to use 
the Lessons in their curriculum and how to discern and respond to students’ 
mathematical misconceptions. 



Teachers understand their role as “facilitator.” To successfully use the Formative 
Assessment Lessons, teachers need to know and understand the non- traditional instructional 
role the tools require. In interviews, both experienced and new teachers across all sites reported 
having a common understanding of their role as a facilitator or coach during the Lesson, which 
is a promising sign of teachers’ knowledge of howto use the tools. They reported that their 
primary role is to facilitate group work by asking guiding questions and move away from the 
more traditional role of providing direct instruction. While all teachers seemed to have a strong 
understanding of what the “facilitator” role entails, some teachers are still adjusting to it or have 
questions about whether direct instruction is appropriate when using the Lessons. 
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Teacher who has adjusted 

I've been teaching for 36 years, and teaching the same way, it's hard to change, teach an old dog new tricks, but 
now that I'm doing it, I love it... .At first, I felt like, God, I'm not teaching! [laughs] But now I realize that they really 
are learning, and doing more on their own. And I don't have to stand up there, teach my heart out, and they just 
looking at me, and still not getting it. But now that they're doing most of the work, they're probably learning more, 
[laughs] (Experienced high school math teacher) 


Teacher who is still adjusting 


My main concern is, how do direct instruction and the Formative Assessment Lessons work together? Because I 
haven't seen a lot of discussion of that. ..When is discovery-type learning appropriate, and when is it maybe not so 
appropriate? And if you're looking at efficiency of transferring information, discovery learning works well when the 
students already have a fair level of expertise in the topic. But when they're novice learners, it's not necessarily the 
most efficient way to move information to them. So that's really something that I think needs to be addressed 
here: when is direct instruction appropriate, and where does it fit in the continuum with the Formative 
Assessment Lesson process? (Experienced high school math teacher) 



In order to use the Formative Assessment Lessons successfully, teachers also have 
to know when to use them in their curriculum. Experienced teachers (96%) reported 
being significantly more knowledgeable about when to use the Formative Assessment Lessons in 
their curriculum compared to new teachers ( 77%) (see Figure 6) . 


MEDIATING CONDITION 


In one site, district leaders and a cohort of teachers worked together to align the Formative Assessment Lessons to their 
curriculum prior to the start of the 2011-12 school year. In this particular site, 98% of teachers reported knowing when 
to use the Formative Assessment Lessons in their curriculum. 


Figure 6. Teachers' Knowledge of how and when to use the Formative Assessment Lessons 


SURVEY ITEM 

I know when to use the Formative Assessment Lessons 
in my curriculum.**** (n=94) 


It was difficult for me to identify students' misconceptions 
from the pre-assessment. (n=94) 


I know how to respond to the mathematical mistakes 
my students make when they are using the 
Formative Assessment Lessons. (n=94) 


Based on the information collected from using the Formative 
Assessment Lessons, I am able to adjust my instruction to 
meet the needs of individual students. (n=94) 
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****Differences between experienced and new teachers are significant at .01 level. 
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Survey and interview data provide somewhat contradictory findings about 

© whether teachers are able to identify students’ mathematical strengths and 

weaknesses. In interviews, teachers reported that by analyzing the pre- assessment and post- 
assessment, they were able to identify gaps in student knowledge and detect growth. However, 
survey data also indicated that one third of teachers (33%) reported experiencing difficulty in 
identifying student misconceptions when reviewing the post- assessments (see Figure 6) . One 
high school math teacher commented on how valuable the pre- assessment is in assessing 
students’ misconceptions: 

When you take a look at this pre-assessment you realize ," Oh my God" - you think they got it and you present 
something like this and then you realize in the in-between there is not the understanding that we expected... The 
pre-assessment definitely tells you where the understandings are or where they are having difficulties so I have to 
be prepared with that. I have to make lists. You have to be prepared that somewhere in the activity the kids are 
going to come up with these issues. You have to be aware and ask them where they are going to go and point out 
where they're going to go with the problem. If there's something that I see that everyone's lacking , I have to 
address it and say ; "Listen guys , / see that we are having difficulties here." So that's how you put the package 
together. (Experienced high school math teacher) 



A strong majority of both experienced and new teachers (84%) reported that they 
know how to adjust their instruction to meet the needs of individual students. In 

interviews, teachers reported using student information about their misconceptions to develop 
feedback questions or re- teach content. 


Teachers develop feedback questions 

I did develop feedback questions ; I dubbed the distance vs. time feedback questions and I basically just went 
through the pretest and found some misconceptions and on the teacher packet, their suggestions for feedback 
questions , those were pretty good , so / did use those too. That helped to develop further learning , it was also the 
only time I taught them anything because the rest was self-discovery. (Experienced high school math teacher) 


Teachers re-teach content 

So then we'll decide basically if we need to do a reteach or not. If we need to spend more time on any topics. ..then 
we'll go back to whichever area they're having trouble with , maybe ask them some of the questions that make 
them think about those ideas a little more. But if they're understanding it, then we can move on to the next thing in 
the curriculum. (Experienced high school math teacher) 
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Conditions 


Classroom Changes 

Robust implementation of MDC brings 
about both changes in teacher beliefs 
and knowledge and changes in the 
classroom, as illustrated in the Theory 
of Action graphic. This section focuses 
on assessing the status of the three 
types of Classroom Changes depicted in 
the Theory of Action and highlighted 
here. 


Teacher Beliefs and Knowledge 


Classroom Changes 



Robust Implementation Indicator: 
Teachers use the tools effectively 
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To understand the status of effective tool use in Year Two of the initiative, we assessed 
teachers’ perceptions of how well they are able to address three central goals of 
using the Formative Assessment Lessons: teaching content, using formative 
assessment, and helping students reach proficiency in the CCSS standards of 
mathematical practice. The Shell Centre’s overview of the Lessons emphasizes the 
importance of math “content described in the [CCSS] standards, focusing on the 
mathematical practices that are the major new challenge in the CCSSM” and 
formative assessment, which “helps teachers and students: 1 ) diagnose difficulties and 
so inform and change teaching; 2) motivate learners by showing them what we value 
and what they still need to learn” ( Shell Centre Website, 20 12) . 


Three Central Goals of Using Formative Assessment Lessons 

• Using Formative Assessment Lessons to teach math content 

• Using Lessons to enhance formative assessment of students 

• Using Lessons to implement the CCSS of mathematical practice 
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Central Goal: Using Formative Assessment Lessons to Teach Math Content 


© Both experienced and new teachers reported that using Lessons has helped them 
find effective strategies for teaching content. As Figure 7 indicates, 78% of experienced 
teachers and 84% of new teachers said that using Lessons is an effective way to teach their 
content. 

Figure 7. Formative Assessment Lessons and Teaching Content 


SURVEY ITEM 

Using the Formative Assessment Lessons has helped 
me find effective strategies for teaching my content. 


| All Teachers Experienced Teachers ^ New Teachers 



© In interviews, teachers reported that using Formative Assessment Lessons helped 
them reinforce content with their students. An Experienced high school Algebra I teacher 
describes how, as a result of the Lesson, helped her students make connections and enrich their 
understanding of the content. 

You get these "aha'sl" everywhere. I was impressed with how many kids did make a system... At the end, when 
making the connection between the table the graph , and the systems , I think that kind of closed some of those 
knowledge gaps for students. 


Central Goal: Using Lessons to Enhance Formative Assessment of Students 



Both experienced and new teachers reported that using Formative Assessment 
Lessons has helped them formatively assess their students. A strong majority of 
teachers ( 79%) reported that using the Lessons has helped them include formative assessment 
in their classes and that using the Lessons has helped them learn detailed informative about 
their students’ mathematical strengths and weakness ( 81 %). There was very little difference 
between experienced and new teachers’ reports on these two survey items (see Figure 8 ). 
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Figure 8. Teachers' reports of Formative Assessment Lessons' impact on their instruction 


Using Formative Assessment Lessons has 
helped me during Lesson instruction 
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Learn new ways to include formative assessment 
in my classes. (n=96) 





Learn detailed information about my students' 
mathematical strengths and weaknesses. (n=96) 






Provide students detailed feedback about their work. 

(n=96) 
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Teachers’ comments during the interviews reflect some of the ways in which they are successfully using 
the Formative Assessment Lessons: 


If you are working with groups, you are constantly going around and monitoring and talking, and that's the most 
important formative assessment you can have. ...Getting around to the groups, having them share out. 
(Experienced high School Math Teacher) 



Almost a quarter of teachers reported that using the Lessons did not help them 
provide students with detailed feedback about their work. This finding indicates that 
teachers need a stronger understanding of how to provide students with feedback during the 
Lesson and after students have completed the post- assessment. 


Central Goal: Using Lessons to Implement the CCSS of Mathematical Practice 


© A strong majority of teachers reported that using the Formative Assessment 

Lessons has helped them implement the Common Core State Standards. There were 
almost no differences between experienced teachers and new teachers’ reports of whether the 
tools helped them implement the CCSS (see Figure 9) . 
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Figure 9. Teachers' reports of Formative Assessment Lesson use and mathematical discourse 


SURVEY ITEM 


Using the Formative Assessment Lessons has 
helped me implement the CCSS. (n=96) 


Using the Formative Assessment Lessons has 
helped me create an environment that promotes 
mathematical discourse. (n=96) 


| All Teachers Experienced Teachers | New Teachers 



5^ Teachers report that using the Formative Assessment Lessons has helped them 
establish a classroom culture that promotes mathematical discourse. One of the 

Standards of mathematical practice posits that students will be able to “attend to precision” or 
that “mathematically proficient students try to communicate precisely to others. They try to use 
dear definitions in discussion with others and in their own reasoning” (CCSS website). 2 This 
finding suggests that two essential components of the Lessons - collaborative and plenary 
activities -are taking root in most teachers’ classrooms (see Figure 9). 


Teachers provided rich descriptions of the mathematical discussions that developed when using the 
Formative Assessment Lessons: 


The students actually talk about math and they are actually having debates and they are debating between who is 
correct. Before , without this type of teaching, they never talked about math , it was always the teacher talking and 
they never got into good discussions or justify their answers and they were never responsible to understand what 
other people were thinking as well. (Experienced high school Math Teacher) 

The Formative Assessment Lessons allowed students to have rich conversations; there was more ownership of their 
learning. Most of my students who are typically quiet and typically don't participate in class , they actually got that 
opportunity to participate. They know that they may not get all the answers correct or they may not finish the 
entire thing , but at least they'll get a good foundation to begin and understand with what I began. (New high 
school Math Teacher) 



Experienced teachers were more likely to report that their use of the Lessons has 
raised their expectations of student work. Eighty percent of experienced teachers 
reported that their use of the Lessons has increased their expectations for students’ 
mathematical work, while only 65% of new teachers reported an increase in their expectations. 
These differences were not significantly different. Teachers with less experience with the 
Lessons may still be in the process of evaluating and understanding the degree to which their 
students are able to engage in these rigorous Lessons, aligned to the CCSS. 


2 http:/ / vww.rorestendards.org/ the- standards/ mathematics/ introduction/ standards- for- mathematical- practice/ 
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Factors That Make a Difference in Effective Tool Use 

Teacher interview data points to seven factors that MDC project leaders need to address to support 
robust implementation of the Formative Assessment Lessons. 


Seven Factors that Make a Difference in Effective Tool Use 


• Access to and alignment of Formative Assessment Lessons 

• Preparing the Formative Assessment Lessons for classroom use 

• Using the Teacher Guide 

• Grouping students for the collaborative activity 

• Differentiation 

• Making time to use the Lessons 


Factor 1: Access to and Alignment of Formative Assessment Lessons 


A central component of using the Formative Assessment Lessons is ensuring that the particular Lesson 
is a good fit for a particular point in the curriculum. In Year One, there were a small number of 
Formative Assessment Lessons to choose from, and district points of contact (POCs) and professional 
development consultants usually chose the Lesson that teachers would use in coordination with the 
professional development. Formative Assessment Lesson use was implemented in this manner so that 
cohorts of teachers could come together for professional development, which often included reviewing 
students’ pre-assessments, the first component of the Lesson. 



In Year Two, access to Formative Assessment Lessons increased. Teacher survey 
results suggest that over a third of teachers (40%) have access to an average of 33 Formative 
Assessment Lessons (median = 23 Lessons). More than a third of teachers surveyed (42%) are 
accessing Lessons from the Shell Centre website and 27% of teachers report that they have 
access to the Lessons from their district’s network share drive. Another 14% of teachers report 
that the Lessons are emailed to them and 12% report that they use hard copies that were 
provided to them during professional development. Unlike in Year One, when teachers 
repeatedly requested more Formative Assessment Lessons during interviews, in Year Two 
teachers did not make this request. 


O In Year Two, district and school leaders are driving the decisions about which 
Formative Assessment Lessons are to be used at a particular point in the 
curriculum. Leaders are making these decisions for two reasons: 

• They have aligned the Formative Assessment Lessons to curriculum pacing guides 
• They have scheduled MDC professional development around Formative Assessment 
Lesson use 


At one site, school leaders decided to align Formative Assessment Lesson use to a new CCSS curriculum 
at certain points during the school year. At another site, district leaders worked with a small group of 
teachers to align Lessons to their new curriculum and pacing guides. While not all sites had developed a 
system of aligning the Lessons to their curriculum, all sites did schedule professional development 
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around Lesson use. In these instances, district or school leaders decided when to use a particular set of 
Lessons over the course of two to three days, and scheduled professional development accordingly. 
During professional development, teachers were often required to complete the pre- assessment with 
their students and then meet after school with their colleagues, district/ school leaders, and professional 
development consultants to facilitate the development of feedback questions. 



By aligning the Formative Assessment Lessons to curriculum guides, district and 
school leaders set expectations for the number of Lessons teachers were required 
to use in Year Two. A strong majority of teachers understand what is expected of 
them. In some instances, specific Formative Assessment Lessons were required, while in other 
sites teachers had the flexibility of choosing which Lesson they used. In interviews, teachers 
reported they were required to use between four to six Formative Assessment Lessons, and at 
the time of the teacher survey, most teachers had used an average of four lessons. A large 
majority of teachers surveyed (88%), as well as interview data, indicated that they understand 
what the district expects of them in implementing the MDC initiative. 



Teachers’ reports of alignment of Formative Assessment Lessons to their 
curriculum and pacing guides have improved, hut the alignment is still not ideal. 

Even though a strong maj ority of teachers ( 8 1%) reported that the Formative Assessment 
Lessons are aligned to their curriculum, 65% reported that they have used a Lesson when it 
wasn’t aligned to their curriculum. Interview data help to explain these slightly contradictory 
results: teachers reported that while the Lessons do align with their curriculum, timing of 
Lesson use was not always perfect. Aligning Formative Assessment Lessons to teachers’ pacing 
guides is very challenging, in light of the guideline to use the Lessons after three-quarters of a 
particular curriculum unit has been taught. Additionally, district and school leaders often 
require participating teachers to use a specific Lesson within the same 2- 3 day window to 
facilitate meaningful professional development around tool use. Yet teachers’ pacing may not be 
the same within the multiple classes they teach, let alone with other teachers in their school 
and/ or district. Below are some quotations from teachers illustrating these differences. 


Excellent alignment 

We've been talking about interior angles ; exterior angles ; in any polygon , and how those worked out. This Lesson 
fits seamlessly with that , so it gave them a chance to use what they know , and some of them remembered it really 
well, some of them didn't , but you get a chance for them to actually show what they knew. And then improve on 
that a little bit more in the next couple of days. (Experienced high school math teacher) 

Good alignment 

It wasn't my decision [to use the Lesson ], it was my department's decision. We weren't sure of the date so... we 
looked at our planning guide and we decided that by that time , we should be more or less on the same page in 
systems. But again , in my case , I was still a little bit behind in a couple of Lessons, but that's why we decided to 
cover equations and finish graphing and we introduced this by graphing and hopefully kids can solve systems using 
different approaches. (Experienced high school math teacher) 
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Teachers are using Lessons in instructionally meaningful ways. Though alignment of 
the Formative Assessment Lessons with curriculum is still a work in progress, teachers are using 
tools in instructionally meaningful ways. For example, teachers used Lessons to: 

• Reinforce previously taught content - 97% 

• Deepen students’ mathematical knowledge - 97% 

• Prepare students for a summative assessment - 77% 

• T each new content - 49% 


O Teachers are using Formative Assessment Lessons to introduce new content. 

Despite the recommendation that Lessons be inserted three-quarters of the way into a particular 
unit, 49% of teachers report using Lessons to introduce new content, indicating that this 
important message about the optimal time to use a Lesson has not reached teachers. 


Factor 2: Preparing the Formative Assessment Lessons for classroom use 


Unlike more traditional mathematics texts and workbooks, the Formative Assessment Lessons are 
dynamic activities that require some upfront preparation for classroom use. This often includes making 
copies of the pre- and post- assessments and cutting and sorting materials for group use. 



Almost half (48%) of the teachers surveyed reported that the preparation required 
for the Lessons is an impediment to using them. In interviews, many teachers noted that 
preparing the Formative Assessment Lessons for use with their students was a considerable 
challenge, especially when the Lesson requires students to use multiple sets of manipulatives: 


Preparing the material! The cutting and the copying and everything. That's the problem , the preparation. And if you 
have three preparations, it will kill you. And especially if you have first period, you have to come in early and print. 
(Experienced high school math teacher) 


The prep was overwhelming but once I had all the materials, I enjoyed it. Yesterday, it took me three hours to cut 
these cards out. ... I feel like if they were already pre-cut, that would be very helpful. Another thing about this is all 
the materials that you have to have ready and prepared, the markers and all, and I feel like I'd like this school to 
provide me with the funding. This is very expensive. (New high school math teacher) 


Factor 3: Formative Assessment Lesson Teacher Guide 


Teachers are supported in their use of the Formative Assessment Lesson with individual Teacher 
Guides (TG) that are included with each Lesson. The TG offers teachers detailed information about how 
to use the Lesson with students - providing instructions, time suggestions to complete each section, 
information about assessing students’ responses on the pre-assessment, common student 
misconceptions, suggestions for teacher responses, and a suggested outline for the entire Lesson. 
Finally, the TG outlines how the particular Formative Assessment Lesson is connected to the CCSS and 
included mathematical goals for the Lesson. 
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A strong majority of teachers surveyed are using the TG and find it helpful. 

Teachers reported that the TG was easy to use, that they adhered to it, and that it helped them 
identify common student misconceptions (see Figure 10). 


Figure 10. Teachers' Perceptions and Use of Lesson Teacher Guide 


SURVEY ITEM 
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The Teacher Guide included with the 


Lessons is easy to use. (n=94) 



1 found the Lesson Teacher Guide helpful in 


identifying common student misconceptions when 
reviewing the pre-assessments.** (n=94) 
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* Differences between experienced and new teachers are significant at .1 level. 


Interview data strongly corroborates survey results. In interviews, most teachers were enthusiastic 
about the TGs. Of 31 teachers who were asked about the usefulness of the guides, 25 were positive, five 
described them as “fairly helpful,” and only one respondent said the TGs were not useful: 

Very Helpful 

Yes , very useful. I really like how they give you suggestions on what to say for the auditory learners and a hand-out 
for the visual learners. I really like how it breaks it down with suggested times , 20 minutes , 5 minutes , it gives me 
the timeframe. - high school math teacher. (Experienced high school math teacher) 

Fairly Helpful 

They're helpful but I find the time estimates too short. You need a lot more time to do it. But , yeah , it's very helpful. 
I like the detail especially with the types of questions you're supposed to be asking because I'm not used to being 
so hands-off so I'm not sure how much I'm allowed to help them , and what types of leading questions I can ask. 
(Experienced high school math teacher) 

Not Helpful 


They have a lot of good leading questions but I feel it needed a little bit more scaffolding. I know that for my 
students , they struggled a lot , and I know that they have a lot of struggles, so a little bit more scaffolding would be 
more helpful. (New high school math teacher) 
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Factor 4: Grouping Students 


Collaborative activity is a central component of Formative Assessment Lessons, requiring students to 
work together in pairs or small groups to complete a portion of the Lesson. Notably, the Teacher Guide 
that accompanies each Lesson does not include guidelines or suggestions on either group composition 
or group size. During a statewide meeting attended primarily by district personnel and a few teachers, 
an MDC professional development consultant advised the audience that students should be grouped 
homogeneously for the collaborative activity. This was the first dear message on grouping strategy in 
two years of numerous MDC professional development sessions, strengthening the suspidon that few 
teachers had received this message. 



Teachers have a strong sense of how to group their students. Teachers’ decisions about 
grouping students are informed by their deep knowledge of students’ personalities and their 
academic strengths and weaknesses. Notably, pairs and groups that proved successful for one 
Formative Assessment Lesson may not be the same mix of students for a subsequent Lesson, 
depending on new classroom and social dynamics and students’ understanding of content, 
underscoring the point that teachers’ decisions about grouping are not static and fixed. 



A strong majority of teachers are purposefully grouping their students. Most 
teachers reported either grouping their students heterogeneously or homogeneously, while a 
little more than quarter of teachers allow their students to choose their own partner! s) during 
the collaborative activity. There are insignificant differences between experienced teachers and 
new teachers’ preferences for grouping students. 


Almost half of teachers (46%) reported grouping their students heterogeneously, because 
they believe that higher and lower level math students can learn from one another by 
working together (see Figure 11): 

[Heterogeneous grouping] worked well because what I found was that the higher level students embraced the 
opportunity to lead other students. ... That was directly helpful to the lower students at the table. (Experienced high 
school math teacher) 


• Twenty-eight percent of teachers surveyed reported that they group their students 
homogenously. In interviews, teachers reported that homogenous grouping (see Figure 11): 
o Helps the teacher reach students more quickly 
o Helps students struggle together and reach a solution 
o Allows high level students to move on 
o Was advised during professional development 
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Figure 11. Decisions about grouping students 
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A majority of both new and experienced teachers preferred to pair students for 
collaborative activities, rather than having students work in groups of three or 
more (see Figure 12). 


Figure 12. Teachers' Grouping of Students for Lessons 



Interview data also suggest that teachers prefer for students to work in pairs. Teachers stated that 
working in pairs increases the accountability for each student and it allows teachers to group students 
heterogeneously or homogenously. The following Experienced teacher reported that he likes to keep the 
group size small: 
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So I'll try and mix these groups as far as ability levels are concern. But I'm also not going to have them bigger than 
about three people. I prefer to keep it at two, actually, because that would improve accountability ... I haven't quite 
made a final decision on two or three students, but I'm leaning toward the two, just so that nobody can sit on the 
side without having to say something or do something with this. 

Factor 5: Differentiating Instruction with the Formative Assessment Lessons 


Meeting the individual needs of diverse learners has always presented a challenge for teachers, but 
there is encouraging news about the accessibility of the Formative Assessment Lessons for different 
student populations. 



Formative Assessment Lessons help teachers provide differentiated instruction to 
their advanced students and students who struggle with math. Avast majority of 
teachers reported that the Formative Assessment Lessons helped them provide differentiated 
instruction to their students who struggle with mathematics (87%) and also to their students 
with advanced mathematical skills (92%) (see Figure 13). However, inside the 92% “all teacher” 
average, there is a 9- point spread between experienced (96%) and new teachers (87%), 
suggesting that with increased time and experience using the Formative Assessment Lessons, 
teachers become more adept at using them to provide differentiation to their more advanced 
students. 


Figure 13. Differentiated lesson instruction for advanced students and struggling students 
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At least a quarter of teachers are experiencing challenges differentiating Lessons 
for special education and ELL students. While most teachers who have special education 
and ELL students in their classes indicated that the Formative Assessment Lessons helped them 
differentiate instruction to both of those student groups, 26% and 30% of all teachers indicated 
that the Lessons were not helping them provide differentiated instruction. This finding suggests 
that even with more experience with the Lessons, a significant minority of teachers do not feel 
that they are helping them provide differentiated instruction to their ELL students and to their 
special education students. 
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Figure 14. Differentiated Lesson instruction for special education and ELL students 


Using Formative Assessment Lessons has helped 
me differentiate during Lesson instruction 

r 

Special Education students 
(n=89) 






ELL students 
(n=54) 



..... 

11 All Teachers Experienced Teachers | New Teachers ( 

) 20% 40% 60% 80% 100% 



Teachers modified the Formative Assessment Lessons to better “fit” the needs of 
their students. Examples of modifications indude: extending or shortening times, strategic 
grouping of students, rephrasing questions, discussing terms and vocabulary, and reading 
instructions to students. In making these modifications, it seems that some teachers are figuring 
out how to differentiate on their own: 


Sometimes the questions [in the Formative Assessment Lessons] are just abstract enough to go over their heads. 
We have a large population of special education kids and they're very concrete thinkers. I'll add something that 
helps them click ; it gives them a hook ; a visual ' something they're familiar with. (Experienced high school math 
teacher) 



Both experienced (90%) and new teachers (86%) reported that the Formative 
Assessment Lessons are flexible enough to fit the needs of the students in their 
classroom. In interviews with teachers, over half of them believe the Formative Assessment 
Lessons meet the needs of the different students they teach: 


For the advanced kids , there's lot of challenges in it and they enjoy the discussion and the interaction. For the 
weaker kids , it's neat when you see those kids trying to make some connections and you do see that. (Experienced 
high school math teacher) 


I noticed yesterday that I have a special needs student who was turning around and helping another group. For him 
to reason and feel comfortable doing that was light years ahead. Fie was helping with a story about the skateboard 
and how it's gradual and continues to be gradual. The other graphs have a stopping point. He kept saying the 
skateboard graph looks like a hill. He changed his language to saying that when it's "curves/' it's "gradual." That 
shows it's changing his thinking. (Experienced high school math teacher) 



Teachers indicated that the Formative Assessment Lessons easily lend themselves 
to differentiated instruction because of their various components. The pre- 
assessment, the collaborated activity, the plenary discussion, and the post- assessment activities 
provide 'built-in 5 opportunities for teachers to analyze students 5 strengths and weaknesses, to 
intentionally pair/ group students, and to purposefully call on particular students during class 
discussions. As two teachers said: 


24 I B r i e f Three: MDC 


I think the Formative Assessment Lessons are designed for students to share knowledge and gain knowledge 
together. I had very little difficulty, and I was anticipating major issues with my ELL students but I had very little 
issues due to their working in groups and pairs. (Experienced high school math teacher) 

That's one of the strengths of these Lessons, is being able to manage, to arrange and design the setting. Within 
designing the setting, you create the hidden strategy of helping that ELL student. (Experienced high school math 
teacher) 



Teachers intentionally grouped their students for the Formative Assessment 
Lessons’ collaborative activities to support differentiated instruction. These findings 
suggest that teachers are using grouping as a way to facilitate differentiated instruction. When 
teachers need to visit many groups in a large class, heterogeneous grouping allows teachers to 
rely on more mathematically advanced students to partner with students who might be 
struggling. With homogeneous grouping, teachers find that they can spend more time with a 
smaller number of groups because the more advanced students need less support than students 
who struggle with math. 



Teachers indicated they would like more professional development on 
differentiating instruction when using the Lessons. While a strong majority of teachers 
surveyed contend that the Formative Assessment Lessons help them provide differentiated 
instruction, 79% also indicated they would welcome professional development on this topic. 
Significantly more Experienced teachers than New teachers requested more professional 
development on differentiating Lesson use. 


Figure 15. Teachers' suggestions for additional professional development 


SURVEY ITEM 

I would like more PD on differentiating Lesson 
use/instruction to better meet students' need** 

(n=96) 
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**Differences between experienced and new teachers are significant at the .1 level. 

In particular, teachers are interested in learning how to provide differentiated instruction to their FT.T, 
and special education students. Said one: 



I've had some issues with modifying them for my kids with lEPs and my ELL kids. Last trimester , 25% of my students 
had lEPs and 10% were ELL , and I had some problems trying to figure out some way to modify it, especially for my 
ELL, and some of them are shutting down completely because a lot of the Formative Assessment Lessons are really, 
really wordy. I had some trouble coming up with new ideas for the ELL kids without changing it too much. 
(Experienced high school math teacher) 



Some teachers are unsure about how much they can adapt the Lessons to meet 
students’ needs. Teachers expressed concerns about how much they can adapt the Formative 
Assessment Lessons to meet the needs of their students without diluting the mathematical 
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richness of the Lessons. This tension between implementation fidelity and making the Lessons 
both accessible and meaningful to diverse students is reflected in the following quotes: 

The challenge is modifying Lessons to make it fit our students' needs and wondering if that's allowed. (New high 
school math teacher) 

I think my ELL students have difficulty, ifind that there is often a connection students can't make and it frustrates 
them and they'll go off -task. It's difficult because I really like the Formative Assessment Lessons' level of vocabulary 
and I don't know how the tool developers can change it to ask it in a more simplified way because they're really 
hitting the content they need to. It's just one of the struggles with ELL in secondary schools because there's so much 
vocabulary they're not comfortable with because they're not exposed to that. Maybe we need a couple of Spanish 
versions to make it more engaging for ELL students? (Experienced high school math teacher) 


Factor 6: Making time for Formative Assessment Lessons 


The issue of time as it relates to teachers’ use of the Formative Assessment Lessons is somewhat 
complex, in that there are several ways teachers consider time in relation to the Lessons. As discussed 
earlier, there is the issue of the time required to prepare the Lesson for classroom use; there are also 
time issues related to how long an individual Lesson takes and how long particular components of the 
Lesson take (or should take according to the Teacher Guide); and the timing of a Lesson within the 
larger curriculum (where to insert it for maximum effectiveness). 



Teachers believe they have the necessary time needed to complete the post- 
assessment. When asked to reflect on the most recent Formative Assessment Lesson used, 
86% of teachers noted that their students had completed the post- assessment, either at the end 
of the plenary discussion (80%) or as homework (6%). Notably, only 4% of teachers indicated 
that they had not given the post- assessment to their students. Of these 4%, half of the teachers 
(only 2) responded that their students had not completed the post- assessment because of time 
constraints. 



Almost a quarter of teachers (22%) reported that their school administrators 
expressed concerns about the time required to use the Formative Assessment 
Lessons. There were notable differences by site on this question: teachers in the two sites with 
more time implementing MDC and/ or district stability were also more likely to report that their 
school administrators had not expressed concerns about the Lessons demanding too much time. 



Teachers’ reports of whether the Lessons take too much time away from covering 
curriculum topics are mixed. Fifty- four percent of teachers surveyed agreed that using the 
Formative Assessment Lessons competes with time to cover required topics, and 46% of 
teachers indicated that the Lessons did not take away too much time. 
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Robust Implementation Indicator: 
Students Exhibit Engagement during Tool Use 


Students exhibit 
engagement 
during tool use 


In interviews, teachers described engagement as: 

• Students can see the big picture, they are giving answers. 

• Students seem to be on task a bit more. 

• Students are really active players during Formative Assessment Lessons. 

• There are more questions being asked. 

• The discussions that Formative Assessment Lessons foster are generally 
pretty rich in mathematics. 


v Almost two-thirds of teachers indicated that their students were more engaged 
when using Formative Assessment Lessons when compared to teachers’ normal 
instruction. An additional 35% of teachers reported that their students found the tools equally 
as engaging as other pedagogical strategies. Only 3% of teachers said the Lessons were less 
engaging to their students (see Figure 16). 


Figure 16. Teachers' perceptions of student engagement compared to their usual instruction 
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Teachers reported that they are ahle to engage a wide variety of students. Survey 
results indicated a strong maj ority of teachers perceive that the Lessons are helping them 
engage a wide variety of students, including those with different mathematical abilities, who 
tend to be disengaged, and students identified as special education. There are very few 
differences between experienced and new teachers on these three survey items ( see Figure 17) . 
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Figure 17. Engaging students with different math abilities, who tend to be disengaged, and who need special education 
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In interviews, both new and experienced teachers noted that their students “enjoy the Formative 
Assessment Lessons,” they like the hands-on aspects, the collaborative activities, and the opportunities 
to participate in discussions. According to one teacher: 

For my lower math students, I find the Formative Assessment Lessons very encouraging to participate. Before, they 
were very passive. Now, they have to be actively participating. They are forced to express some idea. It also 
includes them into thinking, so that's good. (Experienced high school math teacher) 



Fewer teachers reported that Formative Assessment Lessons help engage 
disruptive students and ELL students. Only 63% of New teachers indicated that the 
Formative Assessment Lessons were helping them engage their FT.T. students, which is 
significantly less than 73% of Experienced teachers. More than a quarter of New and 
Experienced teachers reported that the Lessons are not helping them engage students who tend 
to be disruptive (see Figure 18). 


Figure 18. Engaging ELL students and students who tend to be disruptive 



28 | Brief Three: MDC 



Student engagement with the Formative Assessment Lessons can take time. One of 

the more compelling findings from the interview data was the number of teachers who 
commented that student engagement with the Lessons was not necessarily automatic or present 
at the beginning of the Lesson. Rather, teachers noted that student engagement increased as 
they became familiar with their new roles and responsibilities within the various components of 
the Lessons: 


Today when they first saw the activity , most of them said, 7 can't do anything.' And some of them tried a few 
things. ...I had a little bit less participation than I normally do with them but I gave some examples and they were 
starting to move in that direction , get an idea of what was happening and what they should be doing. (Experienced 
high school math teacher) 

They aren't comfortable with their expected roles of asking each other guestions or discussing. ...I think that the 
more we do this , the better the culture will be. It's just difficult to start that way... My most difficult thing is they are 
afraid to fail. They don't struggle and get the wrong answer. They struggle and stop. (Experienced high school math 
teacher) 


This finding suggests that to participate fully in Formative Assessment Lessons, students need to 
understand both their roles and responsibilities within individual Lesson components, as well as 
understanding the roles and responsibilities of their teacher. 


Robust Implementation Indicator: 
Perceptions of Improved Student Learning. 


For the MDC initiative to be sustained, teachers 
need to see evidence that their use of Formative 
Assessment Lessons have a positive impact on 
student learning. In Year One of the evaluation, 
survey evidence of teachers’ perceptions of the 
Lessons’ impact on student learning was strong, but 
the interview data related to student learning was mixed and less 
definitive. In Year Two, results are more consistent: there is both 
strong survey and interview data to suggest that teachers see the 
academic value of using the Lessons and teachers with more 
experience using the Lessons indicated this most frequently. 


Teacher perceive ■ 
of improvement in 
student learning 


© 


UCLA's National Center for 
Research on Evaluation, 
Standards, and Student 
Testing (CRESST) is 
conducting research and 
validation studies to generate 
new knowledge about 
student learning by using 
assessment tasks as outcome 
measures in a study of the 
impact of the MDC initiative. 


Survey and interview data suggest that a large majority of 
teachers have observed academic benefits of using the Formative Assessment 
Lessons with their students, though these reports were more common among Experienced 
teachers than with New teachers (see Figure 19). For example. Experienced teachers’ reports of 
the positive impact that Lesson have on preparing students for the postsecondaiy arena or 
increasing reasoning skills are at least ten percentage points higher on these two survey items 
(see Figure 19). 
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Figure 19. Teachers' perceptions of the Formative Assessment Lessons effect on student learning 
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Formative Assessment Lessons could help students become 
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Formative Assessment Lessons increased students' 
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**Differences between experienced and new teachers are significant at .1 level. 



Interview data about the Lessons’ impact on student learning is much stronger and 
more definitive in Year Two than Year One. Teachers in Year Two reported that the 
Lessons are having a positive impact on student learning in a range of ways, including helping 
students develop new ways of thinking mathematically. When using the Lessons, teachers 
reported that students are: 


• Increasing their content knowledge. A strong majority of teachers (92%) reported that 
after reviewing the post- assessment, they observed growth in student understanding of the 
material (math content). 

There weren't any groups that didn't figure out what they needed to figure out. Every group came to a conclusion 
that was correct. (New high school math teacher) 

I've been able to use the Pythagorean Theorem.../ completed it with my geometry class... I'm very excited because I 
did the post-assessment last week , and all my kids got good grades and they were able to get the areas and draw 
the tilted square. So I was very happy... .Yes [I see growth from the pre-assessment to the post-assessment]. So 
that's something that , / never approach the Pythagorean Theorem that way , and I think it's a great way. 
(Experienced high school math teacher) 


• Reasoning through math problems. In addition to the positive survey data about 
Formative Assessment Lesson impact on students’ reasoning skills in Figure 19, teacher 
interview data also suggest that the Lesson helped increase students’ reasoning skills. 
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I think especially with the time-distance graph, it really helped their reasoning skills, because they all misinterpreted 
the graphs at the beginning. And then by the end. ..the majority really knew how to read the graphs and look and 
see that, this is how, on the graph, if it's going up, that doesn't mean that he's going uphill, it means that his 
distance is increasing. So, I thought that that one really, really helped their reasoning skills and their ability to read 
a graph. I really liked that one. (Experienced high school math teacher) 

• Participating in rich mathematical discussions. In Year One, RFA reported that 

“students are beginning to engage in mathematical discussions” (MDC Teacher Booklet, pg. 12). 
In Year Two, teachers reported that students are engaging in rich mathematical discussions that 
require them to justify and explain their answers to problems. Teachers reported that these 
mathematical discussions are contributing to student learning. 

Another success with this, [is]... the students actually talk about math and they are actually having debates and 
they are debating between who is correct and not and before without this type of teaching, they never talked about 
math, it was always the teacher talking and they never got into good discussions or justify their answers and they 
were never responsible to understand what other people were thinking as well. (Experienced high school math 

teacher) 


• Discovering different methods of solving the same math problem. Teachers reported 
that the Lessons helped students realize that there is more than one way to solve math 
problems. 

It [the Lesson] gave them opportunities to look at things in different representations. I think so often they're used to 
seeing one representation of things. And I think a lot of the Formative Assessment Lessons present the 
mathematical content in multiple representations. (Experienced high school math teacher) 



Students’ comfort level in persevering through difficult math problems and 
adjusting to the new student and teacher roles was mixed. An important goal of the 
CCSS of mathematical practice is that students persevere through difficult problems. In the case 
of the Formative Assessment Lessons, students must learn to persevere while at the same time 
adjusting to the new teacher and student roles. This new role requires teachers to facilitate 
discussions by asking guiding questions instead of teaching through direct instruction or giving 
students specific answers to their questions. Classroom observations of teachers using the 
Formative Assessment Lessons revealed that there was not much consistency or uniformity in 
how teachers introduced the Lessons to their students, how they explained both the roles of 
students and teacher, and teachers’ expectations for students during the different components of 
the Lessons. 


The two quotes from teachers below reflect the teachers’ mixed reports about whether students have 
adjusted to their new roles. 

Some teachers mention that students' ability to persevere through the Lesson was a challenge when they first 
began using the Lessons, but students are beginning to adapt. 
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I see them struggle for an answer instead of giving up. If they don't give up the first time then they have to work 
their way through it and that's working for them and so they are trying to work through it. I tell them if they don't 
have a headache by the time they leave class they haven't done anything yet, so it is strengthening it. They are not 
just stopping the first step. They are working to the second step. They are strengthening and building those 
processes. (Experienced high school math teacher) 


Other teachers reported that students are still adjusting to the new pedagogical approach and are unsure of 
how to persevere through the Formative Assessment Lessons. 

They weren't comfortable with it. They didn't want to say that they didn't know it. If they knew it, they'd feel more 
comfortable with the activity, but they're not used to being let go and just on their own to do something. 
(Experienced high school math teacher) 



Similar to Year One, a few teachers are unsure of the academic benefit of the 
Formative Assessment Lessons. Some are unsure about the Lessons’ impact on reasoning 
skills, while others question the Lessons’ contribution to the learning of content. As the 
comments below reflect, some teachers are too new to the MDC initiative to know if using the 
Lessons are leading to increased content knowledge and reasoning skills, and other teachers are 
eager for empirical evidence 


I have not personally been involved in it long enough to see the pay off ot the end. Hopefully that comes in the 
spring semester , when we start getting into all the assessments. (Experienced high school math teacher) 


I'm looking forward to seeing the results of that. Are the kids moving forward mathematically because of this? And 
just looking at the student work and the Formative Assessment Lessons , it's very difficult to tell how much math 
content they've learned. ..So I don't know. We've got to wait and look at the research. (Experienced high school 
math teacher) 


Intermediate Outcome: Broad and Deep Instructional Change 

An important indicator of the sustainability of an instructional or curricular 
initiative is the degree to which teachers’ instructional practices have changed as 
a result of their participation in the initiative. In Year Two of the MDC initiative. 
Experienced teachers were asked if their level of tool use has increased since Year 
One, and all teachers were asked if the initiative has gained traction in their 
schools and whether their instructional practices had evolved to include MDC- 
type activities when they were not using Formative Assessment Lessons. 

Experienced teachers are significantly more likely to use MDC formative 
assessment strategies than new teachers ( 81 % to s8%)3 (Figure 20) . Interview data also 
indicated that teachers are including MDC instructional strategies throughout the year, 
including formative assessment, giving students “think-time,” and providing class time for 
students to work collaboratively: 



3 Differences between experienced and new teachers are significant at the .001 level. 
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I think the biggest thing for us was those five steps we started with, three years ago in February. Getting that 
formula she [professional development consultant] gave us to respond to who's talking, and lead them into 
questions that move the Lesson along, and trying not to feed them the answer, waiting, because they know more 
math than you think. That stuck with us more than anything else in the program. That's the one thing applied to all- 
I started thinking about now's the time to use the five strategies. That was a keeper. (Experienced high school math 
teacher) 

This has expanded me to do more to work in a group, even more than I have done in the past but I like that a lot... 
We had done a practice ACT test because a lot of them are taking it for the last time, trying to get their scores up so 
I gave it to them for the last time. I scored it over the weekend with no comments on it, handed it back to them 
today and said now work quietly for 30 minutes making your corrections and they did. They worked very quietly, 
now share with your group, let your group discuss the model and they were great discussing among and it's there 
again. I mean if I would have just told them the answer, ok that's great but all of a sudden they were having to 
come up with different strategies and they were seeing well maybe this person did it a little differently than we 
did. (Experienced high school math teacher) 

Experienced and New teachers differed in their perceptions of how the Formative 
Assessment Lessons have impacted their everyday instruction. For example, a higher 
percentage of experienced teachers (47%) than new teachers (37%) reported that their use of 
Lessons had helped them find effective strategies for teaching other content (see Figure 20) . 
However, a higher percentage of new teachers reported that Lessons has helped them create an 
environment that promotes mathematical discourse when teaching other content, and that what 
they learned from using the Lessons has helped them better understand their students’ 
mathematical strengths and weaknesses. Approximately 35-40% of all teachers, both 
Experienced and New, report that their use of Lessons has enriched their non- Lesson teaching 
in the following ways: 

Learn new ways to include formative assessment in my classes 
Learn new ways to include formative assessment in my classes 
Implement the Common Core State Standards 
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Figure 20. Teachers' reports of how Formative Assessment Lessons have impacted everyday instruction 


During non-Lesson instruction, using the Lessons 
has helped me... 

Find effective strategies for teaching my subject content. 

(n=96) 

Create an environment that promotes 
mathematical discourse. (n=96) 

Learn detailed information about my students' 
mathematical strengths and weaknesses. (n=96) 

Learn new ways to include formative assessment 
in my classes. (n=96) 

Provide students with more detailed feedback 
about their work. (n=96) 

Implement the Common Core State Standards. 

(n=96) 
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© Experienced teachers reported that their participation in MDC has increased 

compared to their first year in the initiative. For example, 92% of experienced teachers 
surveyed indicated that they have used Formative Assessment Lessons more often this year than 
last year, and that they have noticed an increase in the number of teachers using the Lessons in 
their school since last year (see Figure 21). 


Figure 21. Experienced teachers’ reports of breadth and depth of change in MDC 
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A strong majority of teachers (85%) reported that the ideas and practices of the 
MDC initiative were gaining traction in their schools (see Figure 22). It is important to 
note that there is very little variation between experienced and new teachers on this item, which 
is a promising sign. Over one- third of teachers (42%) reported that they have shared Formative 
Assessment Lessons with teachers who not participating in the MDC initiative. While a vast 
majority of teachers (93%) reported that their district was committed to sustaining the MDC 
initiative, this perception of district commitment was significantly more common among 
experienced teachers (96%) than new teachers (88%). 
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Figure 22. Teachers' perceptions of traction of the MDC Initiative 
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Teachers identified challenges to the spreading and deepening of tool use in their 
schools. Many teachers have identified some stumbling blocks in the scaling- up of MDC in 
their schools and districts. Almost two- thirds of teachers (62%) reported that there are 
competing curricular initiatives or programs in their district (Figure 22). Teachers also reported 
feeling overwhelmed by new instructional and curricular initiatives in their district. In one 
district in which participation in MDC is voluntary, many teachers opted not to participate 
because of the multiple initiatives in their district. 


Some teachers feel overwhelmed by MDC and other initiatives 

In terms of teachers' enthusiasm for them , I don't know ; it's early , second ’, maybe third year. We are going through 
a lot of changes this year , the standards , the curriculum right now unfortunately it is somewhat of a nuisance in 
that we are trying so hard, doing so much to change all the Lessons we have done , it's putting this on top of this, 
it's hard to look and say oh boy, and so it's not a good time to ask that. (Experienced high school math teacher) 


Some teachers did not participate in MDC due to multiple district initiatives 

So with that optional being there, and with a lot of the other things that have been given to us all at the same time, 
option carries a lot of power. So when it's optional, there aren't going to be a lot of people, especially a lot of 
people who have a lot going on with other things that are going to take advantage of it. (New high school math 
teacher) 


Recommendations 

At the end of Year Two of the MDC Initiative, Experienced teachers exhibit more signs of robust 
implementation than do teachers new to the MDC initiative. They are more likely to report high buy-in 
to initiative principles. They have higher levels of knowledge about tool use and report more positively 
about using the Lessons. They also report higher levels of student engagement and of student learning. 
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This is a promising trend. At least in the early stages of the initiative, increased knowledge of and 
experience with the tools seems to lead to increased confidence in the tools and higher perceptions of 
positive outcomes. As the use of the Formative Assessment Lessons continues to expand, we provide a 
set of recommendations for supporting and sustaining this expansion. 


Support new MDC teachers. 

• Provide targeted support for new MDC teachers. Data indicate that learning to use the 
Formative Assessment Lessons is a developmental process that gets easier as teachers gain more 
experience and confidence. Supporting experienced teachers to work with new teachers in an 
especially promising strategy. 

• Support new teachers in aligning Formative Assessment Lessons to their 
curriculum. Almost a quarter of new teachers reported that they did not know when to use the 
Lessons in their curriculum. 

Help teachers use tools with fidelity. 

• Communicate more clearly the non-negotiable aspects of MDC implementation. 

Teachers expressed uncertainties about which modifications and adaptations are allowed, and 
what changes dilute the mathematical richness of the Lessons. Teachers need support in 
determining what aspects of the Formative Assessment Lessons are negotiable and what aspects 
are fixed. 

• Tool developers and PD providers need to provide clarity around how the tools 
should be used. According tool developers. Formative Assessment Lessons are intended to be 
used after three-quarters of a particular curricular unit has been taught. However, almost half of 
teachers reported using Lessons to introduce new content - minimizing the intended goal of the 
Lessons to provide formative information about student math knowledge. 

Provide continued Professional Learning Opportunities to address key teacher needs 

• Help teachers discern students’ mathematical strengths and weaknesses from the 
pre-assessment. An important process in using the Formative Assessment Lessons involves 
reviewing the pre- assessment to gauge students’ misconceptions of the content so they can 
develop guiding questions to help close students’ knowledge gaps. Almost a third of teachers 
surveyed reported that it was difficult for them to identify their students’ misconceptions on the 
pre- assessment. 

• Formative Assessment Lesson tool developers and professional development providers 
need to continue to increase Initiative capacity to provide training and materials to 
help teachers differentiate and scaffold instruction. Helping teachers address their 
concerns about differentiating Lesson instruction is crucial to robust implementation of the 
tools and to ensuring that they support improved learning outcomes for all students. In 
addition, partners, districts and schools need to provide targeted support and strategies for 
teachers on this issue. 
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• Provide professional development on how to build student engagement when 
using the Formative Assessment Lessons. Teachers would benefit from support and/ or 
professional development on howto better engage both ELL students and students who tend to 
be disruptive during class. Several relatively low-cost ideas include: 

• Classroom visits: teachers visit the classrooms of their colleagues who are successfully 
engaging students during a Lesson. 

• Co- teaching: Invite a teacher who has demonstrated success with engaging students to 
co- teach a Lesson with a teacher who is experiencing challenges. 

• Recruit teachers who have demonstrated success with engaging students to offer 
professional development. 

• Support teachers in developing strategies for communicating to students their roles, 
responsibilities, and expectations during a Formative Assessment Lessons. To participate fully 
in Lessons, and to reap the intellectual benefits, students need to understand the purpose of the 
Lessons, what their roles and responsibilities are during the different components of the Lesson, 
and what is expected of them. When teachers introduce the Lessons to the students, they should 
spend some time communicating the purpose of the Lesson and emphasize the role of both 
students and teachers. In time, as more Formative Assessment Lessons are used, and as 
students become more familiar with them, they will become more comfortable and confident in 
their role. 

• Stress the timing of Formative Assessment Lesson use within a unit. Teachers need 
to uniformly understand that the Formative Assessment Lessons are not designed to introduce 
new content; rather, the Lessons are most effective when introduced and used three-quarters of 
the way into a unit. 


Create the Time and Space for Effective Tool Use. 

• Continue to communicate to teachers about MDC’s relationship with other school 
and district initiatives. While teachers find value in using the Formative Assessment 
Lessons, a significant minority of teachers noted feeling “overwhelmed” by the many initiatives 
in their district. Teachers would benefit from school and district wide discussions on where and 
how the MDC fits with and supports other initiatives. In turn, school and district leaders could 
make sure that teachers understand the ‘big picture,’ thereby reassuring them that their use of 
the Lessons is not only supported and encouraged, but is integral to the school and district’s 
academic mission. 

• While there has been considerable improvement in the area of aligning the 
Formative Assessment Lessons with school curricula, there is still work to be 
done. Using the Lessons in instructionally meaningful ways is critically important to the 
sustainability of the initiative. Once the Lessons have been aligned to the curriculum, teachers 
need some latitude regarding when they can use the Lessons (within the timeframe) based on 
their knowledge of students’ readiness to participate in the Lesson. 
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• Support teachers in preparing the Formative Assessment Lessons for classroom 
use. Preparing Lessons for classroom use is a burdensome task and has the potential to 
threaten teachers’ use of the Lessons. Devise ways to ease this burden for teachers. In one 
district, there was a discussion about purchasing a laminating machine so that teachers could 
laminate materials for subsequent use. These types of solutions will lessen the logistical weight 
of Lessons preparation and give teachers more time to integrate the pedagogical aspects of the 
initiative. 
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